{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6x1ypzczQCwy"
   },
   "source": [
    "# Reading Data from BigQuery with TFX and Vertex AI Pipelines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning objectives\n",
    "\n",
    "* Set up variables.\n",
    "* Create a pipeline.\n",
    "* Run the pipeline on Vertex AI Pipelines."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_VuwrlnvQJ5k"
   },
   "source": [
    "### Introduction\n",
    "\n",
    "In this notebook, you use the BigQueryExampleGen component of [Google Cloud Big Query](https://www.tensorflow.org/tfx/api_docs/python/tfx/v1/extensions/google_cloud_big_query) module that reads data from BigQuery to TensorFlow Extended (TFX) pipelines. This notebook-based tutorial uses [BigQuery](https://cloud.google.com/bigquery) as a data source to train an ML model. The ML pipeline is constructed using TFX and run on Vertex AI Pipelines.\n",
    "\n",
    "This notebook is based on the TFX pipeline you built in [Simple TFX Pipeline for Vertex Pipelines Tutorial](https://www.tensorflow.org/tfx/tutorials/tfx/gcp/vertex_pipelines_simple). If you have not read that tutorial yet, you should read it before proceeding with this notebook.\n",
    "\n",
    "[BigQuery](https://cloud.google.com/bigquery) is a serverless, highly scalable, and cost-effective multi-cloud data warehouse designed for business agility. TFX can be used to read training data from BigQuery and to\n",
    "[publish the trained model](https://www.tensorflow.org/tfx/api_docs/python/tfx/extensions/google_cloud_big_query/pusher/executor/Executor) to BigQuery."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cNA00n3irPgE"
   },
   "source": [
    "## Set up\n",
    "If you have completed\n",
    "[Simple TFX Pipeline for Vertex Pipelines Tutorial](https://www.tensorflow.org/tfx/tutorials/tfx/gcp/vertex_pipelines_simple),\n",
    "you will have a working GCP project and a GCS bucket and that is all you need\n",
    "for this notebook. Please read the preliminary tutorial first if you missed it.\n",
    "\n",
    "__Note__: By default the Vertex AI Pipelines uses the default GCE VM service account of\n",
    "format `[project-number]-compute@developer.gserviceaccount.com`. You need to\n",
    "give a permission to use BigQuery to this account to access BigQuery in the\n",
    "pipeline. Add __BigQuery User__ role to the account.\n",
    "\n",
    "Please see\n",
    "[Vertex documentation](https://cloud.google.com/vertex-ai/docs/pipelines/configure-project)\n",
    "to learn more about service accounts and IAM configuration.\n",
    "\n",
    "Each learning objective will correspond to a _#TODO_ in this student lab notebook -- try to complete this notebook first and then review the [solution notebook](../solutions/reading_data_from_bigquery_with_TFX_and_vertex_pipelines.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WJbPaFzKrPgN"
   },
   "source": [
    "### Install python packages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QVWOEGgMrPgO"
   },
   "source": [
    "You will install required Python packages including TFX and KFP to author ML\n",
    "pipelines and submit jobs to Vertex AI Pipelines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "osJJdvmIrPgP",
    "outputId": "bbc104a2-b2b4-4060-e611-b1364284b992"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pip in /opt/conda/lib/python3.7/site-packages (22.0.4)\n",
      "Collecting pip\n",
      "  Downloading pip-22.1-py3-none-any.whl (2.1 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.1/2.1 MB\u001b[0m \u001b[31m31.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hInstalling collected packages: pip\n",
      "  Attempting uninstall: pip\n",
      "    Found existing installation: pip 22.0.4\n",
      "    Uninstalling pip-22.0.4:\n",
      "      Successfully uninstalled pip-22.0.4\n",
      "Successfully installed pip-22.1\n",
      "Collecting tfx[kfp]<2\n",
      "  Downloading tfx-1.7.1-py3-none-any.whl (2.5 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.5/2.5 MB\u001b[0m \u001b[31m46.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hCollecting google-api-python-client<2,>=1.8\n",
      "  Downloading google_api_python_client-1.12.11-py2.py3-none-any.whl (62 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.1/62.1 kB\u001b[0m \u001b[31m10.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: google-apitools<1,>=0.5 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (0.5.31)\n",
      "Collecting docker<5,>=4.1\n",
      "  Downloading docker-4.4.4-py2.py3-none-any.whl (147 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m147.0/147.0 kB\u001b[0m \u001b[31m21.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: google-cloud-bigquery<3,>=2.26.0 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (2.34.2)\n",
      "Requirement already satisfied: tensorflow-hub<0.13,>=0.9.0 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (0.12.0)\n",
      "Collecting portpicker<2,>=1.3.1\n",
      "  Downloading portpicker-1.5.0-py3-none-any.whl (14 kB)\n",
      "Requirement already satisfied: absl-py<2.0.0,>=0.9 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (0.15.0)\n",
      "Requirement already satisfied: apache-beam[gcp]<3,>=2.36 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (2.37.0)\n",
      "Collecting packaging<21,>=20\n",
      "  Downloading packaging-20.9-py2.py3-none-any.whl (40 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m40.9/40.9 kB\u001b[0m \u001b[31m7.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: keras-tuner<2,>=1.0.4 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (1.1.0)\n",
      "Collecting kubernetes<13,>=10.0.1\n",
      "  Downloading kubernetes-12.0.1-py2.py3-none-any.whl (1.7 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m83.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting attrs<21,>=19.3.0\n",
      "  Downloading attrs-20.3.0-py2.py3-none-any.whl (49 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m49.3/49.3 kB\u001b[0m \u001b[31m8.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: tensorflow-transform<1.8.0,>=1.7.0 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (1.7.0)\n",
      "Requirement already satisfied: numpy<2,>=1.16 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (1.19.5)\n",
      "Collecting tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5\n",
      "  Downloading tensorflow-2.8.0-cp37-cp37m-manylinux2010_x86_64.whl (497.5 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m497.5/497.5 MB\u001b[0m \u001b[31m1.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting ml-metadata<1.8.0,>=1.7.0\n",
      "  Downloading ml_metadata-1.7.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (6.6 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.6/6.6 MB\u001b[0m \u001b[31m97.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mta \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: tensorflow-serving-api!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3,>=1.15 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (2.8.0)\n",
      "Collecting pyarrow<6,>=1\n",
      "  Downloading pyarrow-5.0.0-cp37-cp37m-manylinux2014_x86_64.whl (23.6 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m23.6/23.6 MB\u001b[0m \u001b[31m54.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: protobuf<4,>=3.13 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (3.19.4)\n",
      "Collecting tensorflow-data-validation<1.8.0,>=1.7.0\n",
      "  Downloading tensorflow_data_validation-1.7.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.4 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.4/1.4 MB\u001b[0m \u001b[31m6.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m0:00:01\u001b[0m\n",
      "\u001b[?25hCollecting click<8,>=7\n",
      "  Downloading click-7.1.2-py2.py3-none-any.whl (82 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m82.8/82.8 kB\u001b[0m \u001b[31m792.3 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting pyyaml<6,>=3.12\n",
      "  Downloading PyYAML-5.4.1-cp37-cp37m-manylinux1_x86_64.whl (636 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m636.6/636.6 kB\u001b[0m \u001b[31m1.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: tfx-bsl<1.8.0,>=1.7.0 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (1.7.0)\n",
      "Collecting ml-pipelines-sdk==1.7.1\n",
      "  Downloading ml_pipelines_sdk-1.7.1-py3-none-any.whl (1.3 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m3.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m0:00:01\u001b[0m\n",
      "\u001b[?25hCollecting tensorflow-model-analysis<0.39,>=0.38.0\n",
      "  Downloading tensorflow_model_analysis-0.38.0-py3-none-any.whl (1.8 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.8/1.8 MB\u001b[0m \u001b[31m15.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: jinja2<4,>=2.7.3 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (2.11.3)\n",
      "Requirement already satisfied: grpcio<2,>=1.28.1 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (1.44.0)\n",
      "Requirement already satisfied: google-cloud-aiplatform<2,>=1.6.2 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (1.11.0)\n",
      "Collecting kfp-pipeline-spec<0.2,>=0.1.10\n",
      "  Downloading kfp_pipeline_spec-0.1.14-py3-none-any.whl (18 kB)\n",
      "Collecting kfp<2,>=1.8.5\n",
      "  Downloading kfp-1.8.12.tar.gz (301 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m301.2/301.2 kB\u001b[0m \u001b[31m2.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from absl-py<2.0.0,>=0.9->tfx[kfp]<2) (1.15.0)\n",
      "Requirement already satisfied: pymongo<4.0.0,>=3.8.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (3.12.3)\n",
      "Requirement already satisfied: orjson<4.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (3.6.7)\n",
      "Requirement already satisfied: requests<3.0.0,>=2.24.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (2.27.1)\n",
      "Requirement already satisfied: hdfs<3.0.0,>=2.1.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (2.6.0)\n",
      "Requirement already satisfied: fastavro<2,>=0.23.6 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (1.4.10)\n",
      "Requirement already satisfied: pydot<2,>=1.2.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (1.4.2)\n",
      "Collecting httplib2<0.20.0,>=0.8\n",
      "  Downloading httplib2-0.19.1-py3-none-any.whl (95 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m95.4/95.4 kB\u001b[0m \u001b[31m15.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: cloudpickle<3,>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (2.0.0)\n",
      "Requirement already satisfied: crcmod<2.0,>=1.7 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (1.7)\n",
      "Requirement already satisfied: python-dateutil<3,>=2.8.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (2.8.2)\n",
      "Collecting dill<0.3.2,>=0.3.1.1\n",
      "  Downloading dill-0.3.1.1.tar.gz (151 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m152.0/152.0 kB\u001b[0m \u001b[31m23.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: oauth2client<5,>=2.0.1 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (4.1.3)\n",
      "Requirement already satisfied: proto-plus<2,>=1.7.1 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (1.20.3)\n",
      "Requirement already satisfied: typing-extensions>=3.7.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (3.10.0.2)\n",
      "Requirement already satisfied: pytz>=2018.3 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (2021.3)\n",
      "Collecting google-cloud-bigtable<2,>=0.31.1\n",
      "  Downloading google_cloud_bigtable-1.7.1-py2.py3-none-any.whl (267 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m267.7/267.7 kB\u001b[0m \u001b[31m33.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: google-cloud-dlp<4,>=3.0.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (3.6.2)\n",
      "Requirement already satisfied: google-cloud-pubsublite<2,>=1.2.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (1.4.1)\n",
      "Requirement already satisfied: cachetools<5,>=3.1.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (4.2.4)\n",
      "Collecting google-cloud-vision<2,>=0.38.0\n",
      "  Downloading google_cloud_vision-1.0.1-py2.py3-none-any.whl (435 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m435.1/435.1 kB\u001b[0m \u001b[31m48.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: grpcio-gcp<1,>=0.2.2 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (0.2.2)\n",
      "Collecting google-cloud-language<2,>=1.3.0\n",
      "  Downloading google_cloud_language-1.3.1-py2.py3-none-any.whl (83 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m83.6/83.6 kB\u001b[0m \u001b[31m11.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting google-cloud-core<2,>=0.28.1\n",
      "  Downloading google_cloud_core-1.7.2-py2.py3-none-any.whl (28 kB)\n",
      "Requirement already satisfied: google-cloud-recommendations-ai<=0.2.0,>=0.1.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (0.2.0)\n",
      "Collecting google-cloud-datastore<2,>=1.8.0\n",
      "  Downloading google_cloud_datastore-1.15.4-py2.py3-none-any.whl (134 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.2/134.2 kB\u001b[0m \u001b[31m21.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: google-cloud-bigquery-storage>=2.6.3 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (2.13.0)\n",
      "Requirement already satisfied: google-cloud-pubsub<3,>=2.1.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (2.11.0)\n",
      "Requirement already satisfied: google-auth<3,>=1.18.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (1.35.0)\n",
      "Collecting google-cloud-spanner<2,>=1.13.0\n",
      "  Downloading google_cloud_spanner-1.19.2-py2.py3-none-any.whl (255 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m255.5/255.5 kB\u001b[0m \u001b[31m36.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting google-cloud-videointelligence<2,>=1.8.0\n",
      "  Downloading google_cloud_videointelligence-1.16.2-py2.py3-none-any.whl (183 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m183.9/183.9 kB\u001b[0m \u001b[31m27.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: websocket-client>=0.32.0 in /opt/conda/lib/python3.7/site-packages (from docker<5,>=4.1->tfx[kfp]<2) (1.3.1)\n",
      "Requirement already satisfied: google-auth-httplib2>=0.0.3 in /opt/conda/lib/python3.7/site-packages (from google-api-python-client<2,>=1.8->tfx[kfp]<2) (0.1.0)\n",
      "Collecting uritemplate<4dev,>=3.0.0\n",
      "  Downloading uritemplate-3.0.1-py2.py3-none-any.whl (15 kB)\n",
      "Requirement already satisfied: google-api-core<3dev,>=1.21.0 in /opt/conda/lib/python3.7/site-packages (from google-api-python-client<2,>=1.8->tfx[kfp]<2) (2.5.0)\n",
      "Requirement already satisfied: fasteners>=0.14 in /opt/conda/lib/python3.7/site-packages (from google-apitools<1,>=0.5->tfx[kfp]<2) (0.17.3)\n",
      "Requirement already satisfied: google-cloud-storage<3.0.0dev,>=1.32.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform<2,>=1.6.2->tfx[kfp]<2) (2.2.1)\n",
      "Requirement already satisfied: google-resumable-media<3.0dev,>=0.6.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigquery<3,>=2.26.0->tfx[kfp]<2) (2.3.2)\n",
      "Requirement already satisfied: MarkupSafe>=0.23 in /opt/conda/lib/python3.7/site-packages (from jinja2<4,>=2.7.3->tfx[kfp]<2) (2.0.1)\n",
      "Requirement already satisfied: scipy in /opt/conda/lib/python3.7/site-packages (from keras-tuner<2,>=1.0.4->tfx[kfp]<2) (1.7.3)\n",
      "Requirement already satisfied: tensorboard in /opt/conda/lib/python3.7/site-packages (from keras-tuner<2,>=1.0.4->tfx[kfp]<2) (2.6.0)\n",
      "Requirement already satisfied: ipython in /opt/conda/lib/python3.7/site-packages (from keras-tuner<2,>=1.0.4->tfx[kfp]<2) (7.32.0)\n",
      "Requirement already satisfied: kt-legacy in /opt/conda/lib/python3.7/site-packages (from keras-tuner<2,>=1.0.4->tfx[kfp]<2) (1.0.4)\n",
      "Collecting google-cloud-storage<3.0.0dev,>=1.32.0\n",
      "  Downloading google_cloud_storage-1.44.0-py2.py3-none-any.whl (106 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m106.8/106.8 kB\u001b[0m \u001b[31m17.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting requests-toolbelt<1,>=0.8.0\n",
      "  Downloading requests_toolbelt-0.9.1-py2.py3-none-any.whl (54 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m54.3/54.3 kB\u001b[0m \u001b[31m10.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting kfp-server-api<2.0.0,>=1.1.2\n",
      "  Downloading kfp-server-api-1.8.1.tar.gz (54 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m54.2/54.2 kB\u001b[0m \u001b[31m10.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hCollecting jsonschema<4,>=3.0.1\n",
      "  Downloading jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m56.3/56.3 kB\u001b[0m \u001b[31m10.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting tabulate<1,>=0.8.6\n",
      "  Downloading tabulate-0.8.9-py3-none-any.whl (25 kB)\n",
      "Collecting Deprecated<2,>=1.2.7\n",
      "  Downloading Deprecated-1.2.13-py2.py3-none-any.whl (9.6 kB)\n",
      "Collecting strip-hints<1,>=0.1.8\n",
      "  Downloading strip-hints-0.1.10.tar.gz (29 kB)\n",
      "  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hCollecting docstring-parser<1,>=0.7.3\n",
      "  Downloading docstring_parser-0.14.1-py3-none-any.whl (33 kB)\n",
      "Collecting fire<1,>=0.3.1\n",
      "  Downloading fire-0.4.0.tar.gz (87 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m87.7/87.7 kB\u001b[0m \u001b[31m14.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: pydantic<2,>=1.8.2 in /opt/conda/lib/python3.7/site-packages (from kfp<2,>=1.8.5->tfx[kfp]<2) (1.9.0)\n",
      "Collecting typer<1.0,>=0.3.2\n",
      "  Downloading typer-0.4.1-py3-none-any.whl (27 kB)\n",
      "Requirement already satisfied: certifi>=14.05.14 in /opt/conda/lib/python3.7/site-packages (from kubernetes<13,>=10.0.1->tfx[kfp]<2) (2021.10.8)\n",
      "Requirement already satisfied: urllib3>=1.24.2 in /opt/conda/lib/python3.7/site-packages (from kubernetes<13,>=10.0.1->tfx[kfp]<2) (1.26.8)\n",
      "Requirement already satisfied: requests-oauthlib in /opt/conda/lib/python3.7/site-packages (from kubernetes<13,>=10.0.1->tfx[kfp]<2) (1.3.1)\n",
      "Requirement already satisfied: setuptools>=21.0.0 in /opt/conda/lib/python3.7/site-packages (from kubernetes<13,>=10.0.1->tfx[kfp]<2) (59.8.0)\n",
      "Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging<21,>=20->tfx[kfp]<2) (3.0.7)\n",
      "Requirement already satisfied: psutil in /opt/conda/lib/python3.7/site-packages (from portpicker<2,>=1.3.1->tfx[kfp]<2) (5.9.0)\n",
      "Requirement already satisfied: keras-preprocessing>=1.1.1 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx[kfp]<2) (1.1.2)\n",
      "Requirement already satisfied: gast>=0.2.1 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx[kfp]<2) (0.4.0)\n",
      "Requirement already satisfied: google-pasta>=0.1.1 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx[kfp]<2) (0.2.0)\n",
      "Collecting keras<2.9,>=2.8.0rc0\n",
      "  Downloading keras-2.8.0-py2.py3-none-any.whl (1.4 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.4/1.4 MB\u001b[0m \u001b[31m68.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: h5py>=2.9.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx[kfp]<2) (3.1.0)\n",
      "Collecting tensorflow-io-gcs-filesystem>=0.23.1\n",
      "  Downloading tensorflow_io_gcs_filesystem-0.25.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (2.1 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.1/2.1 MB\u001b[0m \u001b[31m86.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: opt-einsum>=2.3.2 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx[kfp]<2) (3.3.0)\n",
      "Collecting tensorboard\n",
      "  Downloading tensorboard-2.8.0-py3-none-any.whl (5.8 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.8/5.8 MB\u001b[0m \u001b[31m99.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: flatbuffers>=1.12 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx[kfp]<2) (1.12)\n",
      "Collecting tf-estimator-nightly==2.8.0.dev2021122109\n",
      "  Downloading tf_estimator_nightly-2.8.0.dev2021122109-py2.py3-none-any.whl (462 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m462.5/462.5 kB\u001b[0m \u001b[31m51.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting numpy<2,>=1.16\n",
      "  Downloading numpy-1.21.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m15.7/15.7 MB\u001b[0m \u001b[31m74.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: wrapt>=1.11.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx[kfp]<2) (1.12.1)\n",
      "Requirement already satisfied: astunparse>=1.6.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx[kfp]<2) (1.6.3)\n",
      "Collecting libclang>=9.0.1\n",
      "  Downloading libclang-14.0.1-py2.py3-none-manylinux1_x86_64.whl (14.5 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m14.5/14.5 MB\u001b[0m \u001b[31m71.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: termcolor>=1.1.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx[kfp]<2) (1.1.0)\n",
      "Collecting pyfarmhash<0.4,>=0.2\n",
      "  Downloading pyfarmhash-0.3.2.tar.gz (99 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m99.9/99.9 kB\u001b[0m \u001b[31m18.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: pandas<2,>=1.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow-data-validation<1.8.0,>=1.7.0->tfx[kfp]<2) (1.3.5)\n",
      "Collecting joblib<0.15,>=0.12\n",
      "  Downloading joblib-0.14.1-py2.py3-none-any.whl (294 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m294.9/294.9 kB\u001b[0m \u001b[31m39.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: tensorflow-metadata<1.8,>=1.7.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow-data-validation<1.8.0,>=1.7.0->tfx[kfp]<2) (1.7.0)\n",
      "Requirement already satisfied: ipywidgets<8,>=7 in /opt/conda/lib/python3.7/site-packages (from tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (7.6.5)\n",
      "Requirement already satisfied: wheel<1.0,>=0.23.0 in /opt/conda/lib/python3.7/site-packages (from astunparse>=1.6.0->tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx[kfp]<2) (0.37.1)\n",
      "Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.52.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core<3dev,>=1.21.0->google-api-python-client<2,>=1.8->tfx[kfp]<2) (1.54.0)\n",
      "Requirement already satisfied: grpcio-status<2.0dev,>=1.33.2 in /opt/conda/lib/python3.7/site-packages (from google-api-core<3dev,>=1.21.0->google-api-python-client<2,>=1.8->tfx[kfp]<2) (1.44.0)\n",
      "Requirement already satisfied: rsa<5,>=3.1.4 in /opt/conda/lib/python3.7/site-packages (from google-auth<3,>=1.18.0->apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (4.8)\n",
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/conda/lib/python3.7/site-packages (from google-auth<3,>=1.18.0->apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (0.2.7)\n",
      "Requirement already satisfied: grpc-google-iam-v1<0.13dev,>=0.12.3 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigtable<2,>=0.31.1->apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (0.12.3)\n",
      "Collecting google-cloud-core<2,>=0.28.1\n",
      "  Downloading google_cloud_core-1.7.1-py2.py3-none-any.whl (28 kB)\n",
      "  Downloading google_cloud_core-1.7.0-py2.py3-none-any.whl (28 kB)\n",
      "  Downloading google_cloud_core-1.6.0-py2.py3-none-any.whl (28 kB)\n",
      "  Downloading google_cloud_core-1.5.0-py2.py3-none-any.whl (27 kB)\n",
      "  Downloading google_cloud_core-1.4.4-py2.py3-none-any.whl (27 kB)\n",
      "  Downloading google_cloud_core-1.4.3-py2.py3-none-any.whl (27 kB)\n",
      "  Downloading google_cloud_core-1.4.2-py2.py3-none-any.whl (26 kB)\n",
      "  Downloading google_cloud_core-1.4.1-py2.py3-none-any.whl (26 kB)\n",
      "INFO: pip is looking at multiple versions of google-cloud-bigtable to determine which version is compatible with other requirements. This could take a while.\n",
      "Collecting google-cloud-bigtable<2,>=0.31.1\n",
      "  Downloading google_cloud_bigtable-1.7.0-py2.py3-none-any.whl (267 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m267.7/267.7 kB\u001b[0m \u001b[31m36.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hINFO: pip is looking at multiple versions of google-cloud-bigquery-storage to determine which version is compatible with other requirements. This could take a while.\n",
      "Collecting google-cloud-bigquery-storage>=2.6.3\n",
      "  Downloading google_cloud_bigquery_storage-2.13.1-py2.py3-none-any.whl (180 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m180.1/180.1 kB\u001b[0m \u001b[31m29.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hINFO: pip is looking at multiple versions of google-auth-httplib2 to determine which version is compatible with other requirements. This could take a while.\n",
      "Collecting google-auth-httplib2>=0.0.3\n",
      "  Downloading google_auth_httplib2-0.1.0-py2.py3-none-any.whl (9.3 kB)\n",
      "INFO: pip is looking at multiple versions of google-auth to determine which version is compatible with other requirements. This could take a while.\n",
      "Collecting google-auth<3,>=1.18.0\n",
      "  Downloading google_auth-1.35.0-py2.py3-none-any.whl (152 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m152.9/152.9 kB\u001b[0m \u001b[31m25.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hINFO: pip is looking at multiple versions of google-api-core[grpc] to determine which version is compatible with other requirements. This could take a while.\n",
      "Collecting google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5\n",
      "  Downloading google_api_core-2.7.3-py3-none-any.whl (114 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.6/114.6 kB\u001b[0m \u001b[31m19.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hINFO: pip is looking at multiple versions of google-api-core to determine which version is compatible with other requirements. This could take a while.\n",
      "  Downloading google_api_core-2.7.2-py3-none-any.whl (114 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.6/114.6 kB\u001b[0m \u001b[31m21.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Downloading google_api_core-2.7.1-py3-none-any.whl (114 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.7/114.7 kB\u001b[0m \u001b[31m19.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Downloading google_api_core-2.7.0-py3-none-any.whl (114 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.7/114.7 kB\u001b[0m \u001b[31m20.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Downloading google_api_core-2.6.1-py3-none-any.whl (114 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.6/114.6 kB\u001b[0m \u001b[31m19.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Downloading google_api_core-2.6.0-py2.py3-none-any.whl (114 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.6/114.6 kB\u001b[0m \u001b[31m18.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Downloading google_api_core-2.5.0-py2.py3-none-any.whl (111 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m111.8/111.8 kB\u001b[0m \u001b[31m18.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hINFO: pip is looking at multiple versions of google-api-core[grpc] to determine which version is compatible with other requirements. This could take a while.\n",
      "  Downloading google_api_core-2.4.0-py2.py3-none-any.whl (111 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m111.7/111.7 kB\u001b[0m \u001b[31m16.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hINFO: pip is looking at multiple versions of google-api-core to determine which version is compatible with other requirements. This could take a while.\n",
      "  Downloading google_api_core-2.3.2-py2.py3-none-any.whl (109 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m109.8/109.8 kB\u001b[0m \u001b[31m17.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Downloading google_api_core-1.31.5-py2.py3-none-any.whl (93 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m93.3/93.3 kB\u001b[0m \u001b[31m15.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: overrides<7.0.0,>=6.0.1 in /opt/conda/lib/python3.7/site-packages (from google-cloud-pubsublite<2,>=1.2.0->apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (6.1.0)\n",
      "Requirement already satisfied: google-crc32c<2.0dev,>=1.0 in /opt/conda/lib/python3.7/site-packages (from google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3,>=2.26.0->tfx[kfp]<2) (1.1.2)\n",
      "Requirement already satisfied: cached-property in /opt/conda/lib/python3.7/site-packages (from h5py>=2.9.0->tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx[kfp]<2) (1.5.2)\n",
      "Requirement already satisfied: docopt in /opt/conda/lib/python3.7/site-packages (from hdfs<3.0.0,>=2.1.0->apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (0.6.2)\n",
      "Collecting pyparsing>=2.0.2\n",
      "  Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m67.8/67.8 kB\u001b[0m \u001b[31m11.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: traitlets>=4.2 in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (5.1.1)\n",
      "Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (3.0.27)\n",
      "Requirement already satisfied: jedi>=0.16 in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (0.18.1)\n",
      "Requirement already satisfied: backcall in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (0.2.0)\n",
      "Requirement already satisfied: pickleshare in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (0.7.5)\n",
      "Requirement already satisfied: decorator in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (5.1.1)\n",
      "Requirement already satisfied: pygments in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (2.11.2)\n",
      "Requirement already satisfied: pexpect>4.3 in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (4.8.0)\n",
      "Requirement already satisfied: matplotlib-inline in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (0.1.3)\n",
      "Requirement already satisfied: nbformat>=4.2.0 in /opt/conda/lib/python3.7/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (5.2.0)\n",
      "Requirement already satisfied: ipykernel>=4.5.1 in /opt/conda/lib/python3.7/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (6.9.2)\n",
      "Requirement already satisfied: widgetsnbextension~=3.5.0 in /opt/conda/lib/python3.7/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (3.5.2)\n",
      "Requirement already satisfied: jupyterlab-widgets>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (1.0.2)\n",
      "Requirement already satisfied: ipython-genutils~=0.2.0 in /opt/conda/lib/python3.7/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (0.2.0)\n",
      "Requirement already satisfied: pyrsistent>=0.14.0 in /opt/conda/lib/python3.7/site-packages (from jsonschema<4,>=3.0.1->kfp<2,>=1.8.5->tfx[kfp]<2) (0.18.1)\n",
      "Requirement already satisfied: importlib-metadata in /opt/conda/lib/python3.7/site-packages (from jsonschema<4,>=3.0.1->kfp<2,>=1.8.5->tfx[kfp]<2) (4.11.3)\n",
      "Requirement already satisfied: pyasn1>=0.1.7 in /opt/conda/lib/python3.7/site-packages (from oauth2client<5,>=2.0.1->apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (0.4.8)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0,>=2.24.0->apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (3.3)\n",
      "Requirement already satisfied: charset-normalizer~=2.0.0 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0,>=2.24.0->apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (2.0.12)\n",
      "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /opt/conda/lib/python3.7/site-packages (from tensorboard->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (1.8.1)\n",
      "Requirement already satisfied: werkzeug>=0.11.15 in /opt/conda/lib/python3.7/site-packages (from tensorboard->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (2.0.3)\n",
      "Requirement already satisfied: markdown>=2.6.8 in /opt/conda/lib/python3.7/site-packages (from tensorboard->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (3.3.6)\n",
      "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /opt/conda/lib/python3.7/site-packages (from tensorboard->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (0.4.6)\n",
      "Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /opt/conda/lib/python3.7/site-packages (from tensorboard->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (0.6.1)\n",
      "Requirement already satisfied: oauthlib>=3.0.0 in /opt/conda/lib/python3.7/site-packages (from requests-oauthlib->kubernetes<13,>=10.0.1->tfx[kfp]<2) (3.2.0)\n",
      "Requirement already satisfied: cffi>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from google-crc32c<2.0dev,>=1.0->google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3,>=2.26.0->tfx[kfp]<2) (1.15.0)\n",
      "Requirement already satisfied: debugpy<2.0,>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (1.5.1)\n",
      "Requirement already satisfied: tornado<7.0,>=4.2 in /opt/conda/lib/python3.7/site-packages (from ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (6.1)\n",
      "Requirement already satisfied: jupyter-client<8.0 in /opt/conda/lib/python3.7/site-packages (from ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (7.1.2)\n",
      "Requirement already satisfied: nest-asyncio in /opt/conda/lib/python3.7/site-packages (from ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (1.5.4)\n",
      "Requirement already satisfied: parso<0.9.0,>=0.8.0 in /opt/conda/lib/python3.7/site-packages (from jedi>=0.16->ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (0.8.3)\n",
      "Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata->jsonschema<4,>=3.0.1->kfp<2,>=1.8.5->tfx[kfp]<2) (3.7.0)\n",
      "Requirement already satisfied: jupyter-core in /opt/conda/lib/python3.7/site-packages (from nbformat>=4.2.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (4.9.2)\n",
      "Requirement already satisfied: typing-utils>=0.0.3 in /opt/conda/lib/python3.7/site-packages (from overrides<7.0.0,>=6.0.1->google-cloud-pubsublite<2,>=1.2.0->apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (0.1.0)\n",
      "Requirement already satisfied: ptyprocess>=0.5 in /opt/conda/lib/python3.7/site-packages (from pexpect>4.3->ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (0.7.0)\n",
      "Requirement already satisfied: wcwidth in /opt/conda/lib/python3.7/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (0.2.5)\n",
      "Requirement already satisfied: notebook>=4.4.1 in /opt/conda/lib/python3.7/site-packages (from widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (6.4.10)\n",
      "Requirement already satisfied: pycparser in /opt/conda/lib/python3.7/site-packages (from cffi>=1.0.0->google-crc32c<2.0dev,>=1.0->google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3,>=2.26.0->tfx[kfp]<2) (2.21)\n",
      "Requirement already satisfied: pyzmq>=13 in /opt/conda/lib/python3.7/site-packages (from jupyter-client<8.0->ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (22.3.0)\n",
      "Requirement already satisfied: entrypoints in /opt/conda/lib/python3.7/site-packages (from jupyter-client<8.0->ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (0.4)\n",
      "Requirement already satisfied: nbconvert>=5 in /opt/conda/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (6.4.4)\n",
      "Requirement already satisfied: argon2-cffi in /opt/conda/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (21.3.0)\n",
      "Requirement already satisfied: Send2Trash>=1.8.0 in /opt/conda/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (1.8.0)\n",
      "Requirement already satisfied: prometheus-client in /opt/conda/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (0.13.1)\n",
      "Requirement already satisfied: terminado>=0.8.3 in /opt/conda/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (0.13.3)\n",
      "Requirement already satisfied: pandocfilters>=1.4.1 in /opt/conda/lib/python3.7/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (1.5.0)\n",
      "Requirement already satisfied: mistune<2,>=0.8.1 in /opt/conda/lib/python3.7/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (0.8.4)\n",
      "Requirement already satisfied: bleach in /opt/conda/lib/python3.7/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (4.1.0)\n",
      "Requirement already satisfied: defusedxml in /opt/conda/lib/python3.7/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (0.7.1)\n",
      "Requirement already satisfied: nbclient<0.6.0,>=0.5.0 in /opt/conda/lib/python3.7/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (0.5.13)\n",
      "Requirement already satisfied: testpath in /opt/conda/lib/python3.7/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (0.6.0)\n",
      "Requirement already satisfied: jupyterlab-pygments in /opt/conda/lib/python3.7/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (0.1.2)\n",
      "Requirement already satisfied: beautifulsoup4 in /opt/conda/lib/python3.7/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (4.10.0)\n",
      "Requirement already satisfied: argon2-cffi-bindings in /opt/conda/lib/python3.7/site-packages (from argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (21.2.0)\n",
      "Requirement already satisfied: soupsieve>1.2 in /opt/conda/lib/python3.7/site-packages (from beautifulsoup4->nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (2.3.1)\n",
      "Requirement already satisfied: webencodings in /opt/conda/lib/python3.7/site-packages (from bleach->nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (0.5.1)\n",
      "Building wheels for collected packages: kfp, dill, fire, kfp-server-api, pyfarmhash, strip-hints\n",
      "  Building wheel for kfp (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for kfp: filename=kfp-1.8.12-py3-none-any.whl size=419048 sha256=a0b0ce59311c80cfe79a542e335a95346fc8c01e6de1739e41906bb4562f5f81\n",
      "  Stored in directory: /home/jupyter/.cache/pip/wheels/54/0c/4a/3fc55077bc88cc17eacaae34c5fd3f6178c1d16d2ee3b0afdf\n",
      "  Building wheel for dill (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for dill: filename=dill-0.3.1.1-py3-none-any.whl size=78544 sha256=12e83d8b305b43346d5ff0e3f6eb88e318a8cf2321f0a4a55a835d2af1e39ea3\n",
      "  Stored in directory: /home/jupyter/.cache/pip/wheels/a4/61/fd/c57e374e580aa78a45ed78d5859b3a44436af17e22ca53284f\n",
      "  Building wheel for fire (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for fire: filename=fire-0.4.0-py2.py3-none-any.whl size=115942 sha256=c02b36ad46df4e3e8da6778489ae758bd4686ccb8d750028b4597d99ceaf09ce\n",
      "  Stored in directory: /home/jupyter/.cache/pip/wheels/8a/67/fb/2e8a12fa16661b9d5af1f654bd199366799740a85c64981226\n",
      "  Building wheel for kfp-server-api (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for kfp-server-api: filename=kfp_server_api-1.8.1-py3-none-any.whl size=95549 sha256=b4a93eb41a6007545a8526f472b102b827e273a065ef5f26ed2f64cb55e4cd9d\n",
      "  Stored in directory: /home/jupyter/.cache/pip/wheels/f5/4e/2e/6795bd3ed456a43652e7de100aca275ec179c9a8dfbcc65626\n",
      "  Building wheel for pyfarmhash (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for pyfarmhash: filename=pyfarmhash-0.3.2-cp37-cp37m-linux_x86_64.whl size=108625 sha256=826b24b95c1e6e67f51e225f8442b9202e5d0369cd24be9dd1eeab908878bde2\n",
      "  Stored in directory: /home/jupyter/.cache/pip/wheels/53/58/7a/3b040f3a2ee31908e3be916e32660db6db53621ce6eba838dc\n",
      "  Building wheel for strip-hints (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for strip-hints: filename=strip_hints-0.1.10-py2.py3-none-any.whl size=22302 sha256=44c57a801ee25d50b59a2bf3d75222588a1b4ef9bc27d117a741ca17fac6e566\n",
      "  Stored in directory: /home/jupyter/.cache/pip/wheels/5e/14/c3/6e44e9b2545f2d570b03f5b6d38c00b7534aa8abb376978363\n",
      "Successfully built kfp dill fire kfp-server-api pyfarmhash strip-hints\n",
      "Installing collected packages: tf-estimator-nightly, tabulate, pyfarmhash, libclang, keras, joblib, uritemplate, tensorflow-io-gcs-filesystem, strip-hints, pyyaml, pyparsing, portpicker, numpy, kfp-pipeline-spec, fire, docstring-parser, dill, Deprecated, click, attrs, typer, requests-toolbelt, pyarrow, packaging, ml-metadata, kfp-server-api, jsonschema, httplib2, docker, kubernetes, google-api-core, tensorboard, google-cloud-core, google-api-python-client, tensorflow, ml-pipelines-sdk, google-cloud-vision, google-cloud-videointelligence, google-cloud-storage, google-cloud-spanner, google-cloud-language, google-cloud-datastore, google-cloud-bigtable, kfp, tensorflow-model-analysis, tensorflow-data-validation, tfx\n",
      "  Attempting uninstall: keras\n",
      "    Found existing installation: keras 2.6.0\n",
      "    Uninstalling keras-2.6.0:\n",
      "      Successfully uninstalled keras-2.6.0\n",
      "  Attempting uninstall: joblib\n",
      "    Found existing installation: joblib 1.0.1\n",
      "    Uninstalling joblib-1.0.1:\n",
      "      Successfully uninstalled joblib-1.0.1\n",
      "  Attempting uninstall: uritemplate\n",
      "    Found existing installation: uritemplate 4.1.1\n",
      "    Uninstalling uritemplate-4.1.1:\n",
      "      Successfully uninstalled uritemplate-4.1.1\n",
      "  Attempting uninstall: pyyaml\n",
      "    Found existing installation: PyYAML 6.0\n",
      "    Uninstalling PyYAML-6.0:\n",
      "      Successfully uninstalled PyYAML-6.0\n",
      "  Attempting uninstall: pyparsing\n",
      "    Found existing installation: pyparsing 3.0.7\n",
      "    Uninstalling pyparsing-3.0.7:\n",
      "      Successfully uninstalled pyparsing-3.0.7\n",
      "  Attempting uninstall: numpy\n",
      "    Found existing installation: numpy 1.19.5\n",
      "    Uninstalling numpy-1.19.5:\n",
      "      Successfully uninstalled numpy-1.19.5\n",
      "  Attempting uninstall: dill\n",
      "    Found existing installation: dill 0.3.4\n",
      "    Uninstalling dill-0.3.4:\n",
      "      Successfully uninstalled dill-0.3.4\n",
      "  Attempting uninstall: click\n",
      "    Found existing installation: click 8.0.4\n",
      "    Uninstalling click-8.0.4:\n",
      "      Successfully uninstalled click-8.0.4\n",
      "  Attempting uninstall: attrs\n",
      "    Found existing installation: attrs 21.4.0\n",
      "    Uninstalling attrs-21.4.0:\n",
      "      Successfully uninstalled attrs-21.4.0\n",
      "  Attempting uninstall: pyarrow\n",
      "    Found existing installation: pyarrow 7.0.0\n",
      "    Uninstalling pyarrow-7.0.0:\n",
      "      Successfully uninstalled pyarrow-7.0.0\n",
      "  Attempting uninstall: packaging\n",
      "    Found existing installation: packaging 21.3\n",
      "    Uninstalling packaging-21.3:\n",
      "      Successfully uninstalled packaging-21.3\n",
      "  Attempting uninstall: jsonschema\n",
      "    Found existing installation: jsonschema 4.4.0\n",
      "    Uninstalling jsonschema-4.4.0:\n",
      "      Successfully uninstalled jsonschema-4.4.0\n",
      "  Attempting uninstall: httplib2\n",
      "    Found existing installation: httplib2 0.20.4\n",
      "    Uninstalling httplib2-0.20.4:\n",
      "      Successfully uninstalled httplib2-0.20.4\n",
      "  Attempting uninstall: docker\n",
      "    Found existing installation: docker 5.0.3\n",
      "    Uninstalling docker-5.0.3:\n",
      "      Successfully uninstalled docker-5.0.3\n",
      "  Attempting uninstall: kubernetes\n",
      "    Found existing installation: kubernetes 23.3.0\n",
      "    Uninstalling kubernetes-23.3.0:\n",
      "      Successfully uninstalled kubernetes-23.3.0\n",
      "  Attempting uninstall: google-api-core\n",
      "    Found existing installation: google-api-core 2.5.0\n",
      "    Uninstalling google-api-core-2.5.0:\n",
      "      Successfully uninstalled google-api-core-2.5.0\n",
      "  Attempting uninstall: tensorboard\n",
      "    Found existing installation: tensorboard 2.6.0\n",
      "    Uninstalling tensorboard-2.6.0:\n",
      "      Successfully uninstalled tensorboard-2.6.0\n",
      "  Attempting uninstall: google-cloud-core\n",
      "    Found existing installation: google-cloud-core 2.2.3\n",
      "    Uninstalling google-cloud-core-2.2.3:\n",
      "      Successfully uninstalled google-cloud-core-2.2.3\n",
      "  Attempting uninstall: google-api-python-client\n",
      "    Found existing installation: google-api-python-client 2.41.0\n",
      "    Uninstalling google-api-python-client-2.41.0:\n",
      "      Successfully uninstalled google-api-python-client-2.41.0\n",
      "  Attempting uninstall: tensorflow\n",
      "    Found existing installation: tensorflow 2.6.3\n",
      "    Uninstalling tensorflow-2.6.3:\n",
      "      Successfully uninstalled tensorflow-2.6.3\n",
      "  Attempting uninstall: google-cloud-vision\n",
      "    Found existing installation: google-cloud-vision 2.7.1\n",
      "    Uninstalling google-cloud-vision-2.7.1:\n",
      "      Successfully uninstalled google-cloud-vision-2.7.1\n",
      "  Attempting uninstall: google-cloud-videointelligence\n",
      "    Found existing installation: google-cloud-videointelligence 2.6.1\n",
      "    Uninstalling google-cloud-videointelligence-2.6.1:\n",
      "      Successfully uninstalled google-cloud-videointelligence-2.6.1\n",
      "  Attempting uninstall: google-cloud-storage\n",
      "    Found existing installation: google-cloud-storage 2.2.1\n",
      "    Uninstalling google-cloud-storage-2.2.1:\n",
      "      Successfully uninstalled google-cloud-storage-2.2.1\n",
      "  Attempting uninstall: google-cloud-spanner\n",
      "    Found existing installation: google-cloud-spanner 3.13.0\n",
      "    Uninstalling google-cloud-spanner-3.13.0:\n",
      "      Successfully uninstalled google-cloud-spanner-3.13.0\n",
      "  Attempting uninstall: google-cloud-language\n",
      "    Found existing installation: google-cloud-language 2.4.1\n",
      "    Uninstalling google-cloud-language-2.4.1:\n",
      "      Successfully uninstalled google-cloud-language-2.4.1\n",
      "  Attempting uninstall: google-cloud-datastore\n",
      "    Found existing installation: google-cloud-datastore 2.5.1\n",
      "    Uninstalling google-cloud-datastore-2.5.1:\n",
      "      Successfully uninstalled google-cloud-datastore-2.5.1\n",
      "  Attempting uninstall: google-cloud-bigtable\n",
      "    Found existing installation: google-cloud-bigtable 2.7.0\n",
      "    Uninstalling google-cloud-bigtable-2.7.0:\n",
      "      Successfully uninstalled google-cloud-bigtable-2.7.0\n",
      "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "tensorflow-io 0.21.0 requires tensorflow<2.7.0,>=2.6.0, but you have tensorflow 2.8.0 which is incompatible.\n",
      "tensorflow-io 0.21.0 requires tensorflow-io-gcs-filesystem==0.21.0, but you have tensorflow-io-gcs-filesystem 0.25.0 which is incompatible.\n",
      "statsmodels 0.13.2 requires packaging>=21.3, but you have packaging 20.9 which is incompatible.\n",
      "pandas-profiling 3.1.0 requires joblib~=1.0.1, but you have joblib 0.14.1 which is incompatible.\n",
      "cloud-tpu-client 0.10 requires google-api-python-client==1.8.0, but you have google-api-python-client 1.12.11 which is incompatible.\n",
      "black 22.1.0 requires click>=8.0.0, but you have click 7.1.2 which is incompatible.\u001b[0m\u001b[31m\n",
      "\u001b[0mSuccessfully installed Deprecated-1.2.13 attrs-20.3.0 click-7.1.2 dill-0.3.1.1 docker-4.4.4 docstring-parser-0.14.1 fire-0.4.0 google-api-core-1.31.5 google-api-python-client-1.12.11 google-cloud-bigtable-1.7.1 google-cloud-core-2.2.2 google-cloud-datastore-1.15.4 google-cloud-language-1.3.1 google-cloud-spanner-1.19.2 google-cloud-storage-2.1.0 google-cloud-videointelligence-1.16.2 google-cloud-vision-1.0.1 httplib2-0.19.1 joblib-0.14.1 jsonschema-3.2.0 keras-2.8.0 kfp-1.8.12 kfp-pipeline-spec-0.1.14 kfp-server-api-1.8.1 kubernetes-12.0.1 libclang-14.0.1 ml-metadata-1.7.0 ml-pipelines-sdk-1.7.1 numpy-1.21.6 packaging-20.9 portpicker-1.5.0 pyarrow-5.0.0 pyfarmhash-0.3.2 pyparsing-2.4.7 pyyaml-5.4.1 requests-toolbelt-0.9.1 strip-hints-0.1.10 tabulate-0.8.9 tensorboard-2.8.0 tensorflow-2.8.0 tensorflow-data-validation-1.7.0 tensorflow-io-gcs-filesystem-0.25.0 tensorflow-model-analysis-0.38.0 tf-estimator-nightly-2.8.0.dev2021122109 tfx-1.7.1 typer-0.4.1 uritemplate-3.0.1\n"
     ]
    }
   ],
   "source": [
    "# Use the latest version of pip.\n",
    "!pip install --upgrade pip\n",
    "!pip install --upgrade \"tfx[kfp]<2\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Restart the kernel\n",
    "\n",
    "Please ignore any incompatibility warnings and errors. **Restart** the kernel to use updated packages. (On the Notebook menu, select Kernel > Restart Kernel > Restart)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "g3pkMt6zrPgQ"
   },
   "source": [
    "Check the package versions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "mvZS3XW2rPgR"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TensorFlow version: 2.8.0\n",
      "TFX version: 1.7.1\n",
      "KFP version: 1.8.12\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "print('TensorFlow version: {}'.format(tf.__version__))\n",
    "from tfx import v1 as tfx\n",
    "print('TFX version: {}'.format(tfx.__version__))\n",
    "import kfp\n",
    "print('KFP version: {}'.format(kfp.__version__))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aDtLdSkvqPHe"
   },
   "source": [
    "### Set up variables\n",
    "\n",
    "You will set up some variables used to customize the pipelines below. Following\n",
    "information is required:\n",
    "\n",
    "* GCP Project id and number. See\n",
    "[Identifying your project id and number](https://cloud.google.com/resource-manager/docs/creating-managing-projects#identifying_projects).\n",
    "* GCP Region to run pipelines. For more information about the regions that\n",
    "Vertex AI Pipelines is available in, see the\n",
    "[Vertex AI locations guide](https://cloud.google.com/vertex-ai/docs/general/locations#feature-availability).\n",
    "* Google Cloud Storage Bucket to store pipeline outputs.\n",
    "\n",
    "**Enter required values in the cell below before running it**.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "EcUseqJaE2XN"
   },
   "outputs": [],
   "source": [
    "GOOGLE_CLOUD_PROJECT = ''         # <--- ENTER THIS\n",
    "GOOGLE_CLOUD_PROJECT_NUMBER = ''  # <--- ENTER THIS\n",
    "GOOGLE_CLOUD_REGION = ''          # <--- ENTER THIS\n",
    "GCS_BUCKET_NAME = ''              # <--- ENTER THIS\n",
    "\n",
    "if not (GOOGLE_CLOUD_PROJECT and  GOOGLE_CLOUD_PROJECT_NUMBER and GOOGLE_CLOUD_REGION and GCS_BUCKET_NAME):\n",
    "    from absl import logging\n",
    "    logging.error('Please set all required parameters.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GAaCPLjgiJrO"
   },
   "source": [
    "Set `gcloud` to use your project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "VkWdxe4TXRHk"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Updated property [core/project].\n"
     ]
    }
   ],
   "source": [
    "!gcloud config set project {GOOGLE_CLOUD_PROJECT}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "CPN6UL5CazNy"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PIPELINE_ROOT: gs://qwiklabs-gcp-00-455aefcf608d/pipeline_root/penguin-bigquery\n"
     ]
    }
   ],
   "source": [
    "PIPELINE_NAME = 'penguin-bigquery'\n",
    "\n",
    "# Path to various pipeline artifact.\n",
    "PIPELINE_ROOT = 'gs://{}/pipeline_root/{}'.format(\n",
    "    GCS_BUCKET_NAME, PIPELINE_NAME)\n",
    "\n",
    "# Paths for users' Python module.\n",
    "MODULE_ROOT = 'gs://{}/pipeline_module/{}'.format(\n",
    "    GCS_BUCKET_NAME, PIPELINE_NAME)\n",
    "\n",
    "# Paths for users' data.\n",
    "DATA_ROOT = # TODO 1: Your code here\n",
    "\n",
    "# This is the path where your model will be pushed for serving.\n",
    "SERVING_MODEL_DIR = 'gs://{}/serving_model/{}'.format(\n",
    "    GCS_BUCKET_NAME, PIPELINE_NAME)\n",
    "\n",
    "print('PIPELINE_ROOT: {}'.format(PIPELINE_ROOT))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nH6gizcpSwWV"
   },
   "source": [
    "## Create a pipeline\n",
    "\n",
    "TFX pipelines are defined using Python APIs as you did in\n",
    "[Simple TFX Pipeline for Vertex Pipelines Tutorial](https://www.tensorflow.org/tfx/tutorials/tfx/gcp/vertex_pipelines_simple).\n",
    "You previously used `CsvExampleGen` which reads data from a CSV file. In this\n",
    "notebook, you will use\n",
    "[`BigQueryExampleGen`](https://www.tensorflow.org/tfx/api_docs/python/tfx/extensions/google_cloud_big_query/example_gen/component/BigQueryExampleGen)\n",
    "component which reads data from BigQuery."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hNg73Slwn8nq"
   },
   "source": [
    "### Prepare BigQuery query\n",
    "\n",
    "You will use the same\n",
    "[Palmer Penguins dataset](https://allisonhorst.github.io/palmerpenguins/articles/intro.html). However, you will read it from a BigQuery table\n",
    "`tfx-oss-public.palmer_penguins.palmer_penguins` which is populated using the\n",
    "same CSV file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "Mb_Kj1U8pBhZ"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Query complete after 0.00s: 100%|██████████| 1/1 [00:00<00:00, 525.67query/s]                          \n",
      "Downloading: 100%|██████████| 5/5 [00:01<00:00,  3.56rows/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>species</th>\n",
       "      <th>culmen_length_mm</th>\n",
       "      <th>culmen_depth_mm</th>\n",
       "      <th>flipper_length_mm</th>\n",
       "      <th>body_mass_g</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0.254545</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.152542</td>\n",
       "      <td>0.291667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0.269091</td>\n",
       "      <td>0.511905</td>\n",
       "      <td>0.237288</td>\n",
       "      <td>0.305556</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0.298182</td>\n",
       "      <td>0.583333</td>\n",
       "      <td>0.389831</td>\n",
       "      <td>0.152778</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0.167273</td>\n",
       "      <td>0.738095</td>\n",
       "      <td>0.355932</td>\n",
       "      <td>0.208333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0.261818</td>\n",
       "      <td>0.892857</td>\n",
       "      <td>0.305085</td>\n",
       "      <td>0.263889</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   species  culmen_length_mm  culmen_depth_mm  flipper_length_mm  body_mass_g\n",
       "0        0          0.254545         0.666667           0.152542     0.291667\n",
       "1        0          0.269091         0.511905           0.237288     0.305556\n",
       "2        0          0.298182         0.583333           0.389831     0.152778\n",
       "3        0          0.167273         0.738095           0.355932     0.208333\n",
       "4        0          0.261818         0.892857           0.305085     0.263889"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%bigquery --project {GOOGLE_CLOUD_PROJECT}\n",
    "SELECT *\n",
    "FROM `tfx-oss-public.palmer_penguins.palmer_penguins`\n",
    "LIMIT 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "arvdbM5jpjNm"
   },
   "source": [
    "All features were already normalized to 0~1 except `species` which is the\n",
    "label. You will build a classification model which predicts the `species` of\n",
    "penguins.\n",
    "\n",
    "`BigQueryExampleGen` requires a query to specify which data to fetch. Because\n",
    "You will use all the fields of all rows in the table, the query is quite simple.\n",
    "You can also specify field names and add `WHERE` conditions as needed according\n",
    "to the\n",
    "[BigQuery Standard SQL syntax](https://cloud.google.com/bigquery/docs/reference/standard-sql/query-syntax)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "7AwysGAVnfJA"
   },
   "outputs": [],
   "source": [
    "QUERY = \"SELECT * FROM `tfx-oss-public.palmer_penguins.palmer_penguins`\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lOjDv93eS5xV"
   },
   "source": [
    "### Write model code.\n",
    "\n",
    "You will use the same model code as in the\n",
    "[Simple TFX Pipeline Tutorial](https://www.tensorflow.org/tfx/tutorials/tfx/penguin_simple)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "aES7Hv5QTDK3"
   },
   "outputs": [],
   "source": [
    "_trainer_module_file = 'penguin_trainer.py'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "Gnc67uQNTDfW"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing penguin_trainer.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile {_trainer_module_file}\n",
    "\n",
    "# Copied from https://www.tensorflow.org/tfx/tutorials/tfx/penguin_simple\n",
    "\n",
    "from typing import List\n",
    "from absl import logging\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "from tensorflow_transform.tf_metadata import schema_utils\n",
    "\n",
    "from tfx import v1 as tfx\n",
    "from tfx_bsl.public import tfxio\n",
    "\n",
    "from tensorflow_metadata.proto.v0 import schema_pb2\n",
    "\n",
    "_FEATURE_KEYS = [\n",
    "    'culmen_length_mm', 'culmen_depth_mm', 'flipper_length_mm', 'body_mass_g'\n",
    "]\n",
    "_LABEL_KEY = 'species'\n",
    "\n",
    "_TRAIN_BATCH_SIZE = 20\n",
    "_EVAL_BATCH_SIZE = 10\n",
    "\n",
    "# Since you're not generating or creating a schema, you will instead create\n",
    "# a feature spec.  Since there are a fairly small number of features this is\n",
    "# manageable for this dataset.\n",
    "_FEATURE_SPEC = {\n",
    "    **{\n",
    "        feature: tf.io.FixedLenFeature(shape=[1], dtype=tf.float32)\n",
    "           for feature in _FEATURE_KEYS\n",
    "       },\n",
    "    _LABEL_KEY: tf.io.FixedLenFeature(shape=[1], dtype=tf.int64)\n",
    "}\n",
    "\n",
    "\n",
    "def _input_fn(file_pattern: List[str],\n",
    "              data_accessor: tfx.components.DataAccessor,\n",
    "              schema: schema_pb2.Schema,\n",
    "              batch_size: int) -> tf.data.Dataset:\n",
    "  \"\"\"Generates features and label for training.\n",
    "\n",
    "  Args:\n",
    "    file_pattern: List of paths or patterns of input tfrecord files.\n",
    "    data_accessor: DataAccessor for converting input to RecordBatch.\n",
    "    schema: schema of the input data.\n",
    "    batch_size: representing the number of consecutive elements of returned\n",
    "      dataset to combine in a single batch\n",
    "\n",
    "  Returns:\n",
    "    A dataset that contains (features, indices) tuple where features is a\n",
    "      dictionary of Tensors, and indices is a single Tensor of label indices.\n",
    "  \"\"\"\n",
    "  return data_accessor.tf_dataset_factory(\n",
    "      file_pattern,\n",
    "      tfxio.TensorFlowDatasetOptions(\n",
    "          batch_size=batch_size, label_key=_LABEL_KEY),\n",
    "      schema=schema).repeat()\n",
    "\n",
    "\n",
    "def _make_keras_model() -> tf.keras.Model:\n",
    "  \"\"\"Creates a DNN Keras model for classifying penguin data.\n",
    "\n",
    "  Returns:\n",
    "    A Keras Model.\n",
    "  \"\"\"\n",
    "  # The model below is built with Functional API, please refer to\n",
    "  # https://www.tensorflow.org/guide/keras/overview for all API options.\n",
    "  inputs = [keras.layers.Input(shape=(1,), name=f) for f in _FEATURE_KEYS]\n",
    "  d = keras.layers.concatenate(inputs)\n",
    "  for _ in range(2):\n",
    "    d = keras.layers.Dense(8, activation='relu')(d)\n",
    "  outputs = keras.layers.Dense(3)(d)\n",
    "\n",
    "  model = keras.Model(inputs=inputs, outputs=outputs)\n",
    "  model.compile(\n",
    "      optimizer=keras.optimizers.Adam(1e-2),\n",
    "      loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
    "      metrics=[keras.metrics.SparseCategoricalAccuracy()])\n",
    "\n",
    "  model.summary(print_fn=logging.info)\n",
    "  return model\n",
    "\n",
    "\n",
    "# TFX Trainer will call this function.\n",
    "def run_fn(fn_args: tfx.components.FnArgs):\n",
    "  \"\"\"Train the model based on given args.\n",
    "\n",
    "  Args:\n",
    "    fn_args: Holds args used to train the model as name/value pairs.\n",
    "  \"\"\"\n",
    "\n",
    "  # This schema is usually either an output of SchemaGen or a manually-curated\n",
    "  # version provided by pipeline author. A schema can also derived from TFT\n",
    "  # graph if a Transform component is used. In the case when either is missing,\n",
    "  # `schema_from_feature_spec` could be used to generate schema from very simple\n",
    "  # feature_spec, but the schema returned would be very primitive.\n",
    "  schema = schema_utils.schema_from_feature_spec(_FEATURE_SPEC)\n",
    "\n",
    "  train_dataset = _input_fn(\n",
    "      fn_args.train_files,\n",
    "      fn_args.data_accessor,\n",
    "      schema,\n",
    "      batch_size=_TRAIN_BATCH_SIZE)\n",
    "  eval_dataset = _input_fn(\n",
    "      fn_args.eval_files,\n",
    "      fn_args.data_accessor,\n",
    "      schema,\n",
    "      batch_size=_EVAL_BATCH_SIZE)\n",
    "\n",
    "  model = _make_keras_model()\n",
    "  model.fit(\n",
    "      train_dataset,\n",
    "      steps_per_epoch=fn_args.train_steps,\n",
    "      validation_data=eval_dataset,\n",
    "      validation_steps=fn_args.eval_steps)\n",
    "\n",
    "  # The result of the training should be saved in `fn_args.serving_model_dir`\n",
    "  # directory.\n",
    "  model.save(fn_args.serving_model_dir, save_format='tf')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-LsYx8MpYvPv"
   },
   "source": [
    "Copy the module file to GCS which can be accessed from the pipeline components.\n",
    "Because model training happens on GCP, you need to upload this model definition.\n",
    "\n",
    "Otherwise, you might want to build a container image including the module file\n",
    "and use the image to run the pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "rMMs5wuNYAbc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Copying file://penguin_trainer.py [Content-Type=text/x-python]...\n",
      "/ [1 files][  3.8 KiB/  3.8 KiB]                                                \n",
      "Operation completed over 1 objects/3.8 KiB.                                      \n"
     ]
    }
   ],
   "source": [
    "!gcloud storage cp {_trainer_module_file} {MODULE_ROOT}/"   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "w3OkNz3gTLwM"
   },
   "source": [
    "### Write a pipeline definition\n",
    "\n",
    "You will define a function to create a TFX pipeline. You need to use\n",
    "`BigQueryExampleGen` which takes `query` as an argument. One more change from\n",
    "the previous notebook is that you need to pass `beam_pipeline_args` which is\n",
    "passed to components when they are executed. You will use `beam_pipeline_args`\n",
    "to pass additional parameters to BigQuery.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "M49yYVNBTPd4"
   },
   "outputs": [],
   "source": [
    "from typing import List, Optional\n",
    "\n",
    "def _create_pipeline(pipeline_name: str, pipeline_root: str, query: str,\n",
    "                     module_file: str, serving_model_dir: str,\n",
    "                     beam_pipeline_args: Optional[List[str]],\n",
    "                     ) -> tfx.dsl.Pipeline:\n",
    "  \"\"\"Creates a TFX pipeline using BigQuery.\"\"\"\n",
    "\n",
    "  # NEW: Query data in BigQuery as a data source.\n",
    "  example_gen = # TODO 2: Your code here\n",
    "\n",
    "  # Uses user-provided Python function that trains a model.\n",
    "  trainer = tfx.components.Trainer(\n",
    "      module_file=module_file,\n",
    "      examples=example_gen.outputs['examples'],\n",
    "      train_args=tfx.proto.TrainArgs(num_steps=100),\n",
    "      eval_args=tfx.proto.EvalArgs(num_steps=5))\n",
    "\n",
    "  # Pushes the model to a file destination.\n",
    "  pusher = tfx.components.Pusher(\n",
    "      model=trainer.outputs['model'],\n",
    "      push_destination=tfx.proto.PushDestination(\n",
    "          filesystem=tfx.proto.PushDestination.Filesystem(\n",
    "              base_directory=serving_model_dir)))\n",
    "\n",
    "  components = [\n",
    "      example_gen,\n",
    "      trainer,\n",
    "      pusher,\n",
    "  ]\n",
    "\n",
    "  return tfx.dsl.Pipeline(\n",
    "      pipeline_name=pipeline_name,\n",
    "      pipeline_root=pipeline_root,\n",
    "      components=components,\n",
    "      # NEW: `beam_pipeline_args` is required to use BigQueryExampleGen.\n",
    "      beam_pipeline_args=beam_pipeline_args)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mJbq07THU2GV"
   },
   "source": [
    "## Run the pipeline on Vertex AI Pipelines.\n",
    "\n",
    "You use Vertex AI Pipelines to run the pipeline as you did in\n",
    "[Simple TFX Pipeline for Vertex Pipelines Tutorial](https://www.tensorflow.org/tfx/tutorials/tfx/gcp/vertex_pipelines_simple).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7mp0AkmrPdUb"
   },
   "source": [
    "You also need to pass `beam_pipeline_args` for the BigQueryExampleGen. It\n",
    "includes configs like the name of the GCP project and the temporary storage for\n",
    "the BigQuery execution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "fAtfOZTYWJu-"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# You need to pass some GCP related configs to BigQuery. This is currently done\n",
    "# using `beam_pipeline_args` parameter.\n",
    "BIG_QUERY_WITH_DIRECT_RUNNER_BEAM_PIPELINE_ARGS = [\n",
    "   '--project=' + GOOGLE_CLOUD_PROJECT,\n",
    "   '--temp_location=' + os.path.join('gs://', GCS_BUCKET_NAME, 'tmp'),\n",
    "   ]\n",
    "\n",
    "PIPELINE_DEFINITION_FILE = PIPELINE_NAME + '_pipeline.json'\n",
    "\n",
    "runner = tfx.orchestration.experimental.KubeflowV2DagRunner(\n",
    "    config=tfx.orchestration.experimental.KubeflowV2DagRunnerConfig(),\n",
    "    output_filename=PIPELINE_DEFINITION_FILE)\n",
    "_ = runner.run(\n",
    "    _create_pipeline(\n",
    "        pipeline_name=PIPELINE_NAME,\n",
    "        pipeline_root=PIPELINE_ROOT,\n",
    "        query=QUERY,\n",
    "        module_file=os.path.join(MODULE_ROOT, _trainer_module_file),\n",
    "        serving_model_dir=SERVING_MODEL_DIR,\n",
    "        beam_pipeline_args=BIG_QUERY_WITH_DIRECT_RUNNER_BEAM_PIPELINE_ARGS))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fWyITYSDd8w4"
   },
   "source": [
    "The generated definition file can be submitted using kfp client."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "tI71jlEvWMV7"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:Creating PipelineJob\n",
      "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob created. Resource name: projects/311388853521/locations/us-central1/pipelineJobs/penguin-bigquery-20220516122323\n",
      "INFO:google.cloud.aiplatform.pipeline_jobs:To use this PipelineJob in another session:\n",
      "INFO:google.cloud.aiplatform.pipeline_jobs:pipeline_job = aiplatform.PipelineJob.get('projects/311388853521/locations/us-central1/pipelineJobs/penguin-bigquery-20220516122323')\n",
      "INFO:google.cloud.aiplatform.pipeline_jobs:View Pipeline Job:\n",
      "https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/penguin-bigquery-20220516122323?project=311388853521\n"
     ]
    }
   ],
   "source": [
    "# docs_infra: no_execute\n",
    "from google.cloud import aiplatform\n",
    "from google.cloud.aiplatform import pipeline_jobs\n",
    "import logging\n",
    "logging.getLogger().setLevel(logging.INFO)\n",
    "\n",
    "aiplatform.init(project=GOOGLE_CLOUD_PROJECT, location=GOOGLE_CLOUD_REGION)\n",
    "\n",
    "job = pipeline_jobs.PipelineJob(template_path=PIPELINE_DEFINITION_FILE,\n",
    "                                display_name=PIPELINE_NAME)\n",
    "# Submit the job\n",
    "# TODO 3: Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "L3k9f5IVQXcQ"
   },
   "source": [
    "Now you can visit the link in the output above or visit 'Vertex AI > Pipelines'\n",
    "in [Google Cloud Console](https://console.cloud.google.com/) to see the\n",
    "progress."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [
    "pknVo1kM2wI2"
   ],
   "name": "Reading data from BigQuery with TFX and Vertex AI Pipelines",
   "provenance": [],
   "toc_visible": true
  },
  "environment": {
   "kernel": "python3",
   "name": "tf2-gpu.2-6.m91",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-6:m91"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
